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Overview 

Last year, MENTOR released the National Agenda for Action: How to Close America's 
Mentoring Gap. Representing the collective wisdom of the mentoring field, the Agenda 
articulates five key strategies and action items necessary to move the field forward and 
truly close the mentoring gap. In an effort to address one of these critical strategies — 
elevating the role of research — MENTOR created the Research and Policy Council, an 
advisory group composed of the nation's leading mentoring researchers, policymakers, 
and practitioners. 

In September 2006, MENTOR convened the first meeting of the Research and Policy 
Council with the goal of increasing the connection and exchange of ideas among 
practitioners, policymakers, and researchers to strengthen the practice of youth mentor- 
ing. The Research in Action series is the first product to evolve from the work of the 
Council — taking current mentoring research and translating it into useful, user-friendly 
materials for mentoring practitioners. 

With research articles written by leading scholars, the series includes ten issues on some 
of the most pressing topics facing the youth mentoring field: 

Issue 1 : Mentoring: A Key Resource for Promoting Positive Youth Development 

Issue 2: Effectiveness of Mentoring Program Practices 

Issue 3: Program Staff in Youth Mentoring Programs: Qualifications, Training, 

and Retention 

Issue 4: Fostering Close and Effective Relationships in Youth Mentoring Programs 

Issue 5: Why Youth Mentoring Relationships End 

Issue 6: School-Based Mentoring 

Issue 7: Cross-Age Peer Mentoring 

Issue 8: Mentoring Across Generations: Engaging Age 50+ Adults as Mentors 

Issue 9: Youth Mentoring: Do Race and Ethnicity Really Matter? 

Issue 10: Mentoring: A Promising Intervention for Children of Prisoners 



RESEARCH IN ACTION, ISSUE 2 I 01 



Using the Series 

Each issue in the series is designed to make the scholarly research accessible to 
and relevant for practitioners and is composed of three sections: 

1 . Research: a peer-reviewed article, written by a leading researcher, summarizing 
the latest research available on the topic and its implications for the field; 

2. Action: a tool, activity, template, or resource, created by MENTOR, with concrete 
suggestions on how practitioners can incorporate the research findings into 
mentoring programs; and 

3. Resources: a list of additional resources on the topic for further research. 



As you read the series, we invite you to study each section and consider what you can 
do to effectively link mentoring research with program practice. Please join us in thank- 
ing the executive editor, Dr. Jean Rhodes, and the author of this issue, Dr. David Dubois, 
for graciously contributing their time and expertise to this project. 






Gail Manza 
Executive Director 



Tonya Wiley 
Senior Vice President 



Cindy Sturtevant Borden 
Vice President 
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RESEARCH 



Effectiveness of Mentoring Program Practices 

David L. DuBois, Ph.D., University of Illinois at Chicago 



Introduction 

At a fundamental level, the central objective of any youth mentoring program is to establish 
and support beneficial relationships between youth and their identified mentors. To be 
considered a high-quality program, a mentoring program arguably needs to accomplish 
this objective in ways that are not only effective (i.e., relationships are established that 
are high in quality and that yield positive impacts on youth outcomes), but also safe (i.e., 
no youth are inadvertently harmed), efficient (i.e., resources are used in a cost-effec- 
tive manner), and sustainable/growth-oriented (i.e., the program is able to continue to 
operate over an extended period of time and serve increasing numbers of youth). This is 
no small feat, to be sure. Mentors and youth typically have not met previously, mentors 
are usually volunteers with no specialized prior training or experience, youth often come 
from highly challenged home and community environments and may exhibit significant 
personal vulnerabilities, and programs in most cases are operating under the constraints 
of limited resources. These considerations underscore the importance of determining 
what approaches, or program practices, are most successful in promoting effectiveness 
and overall program quality. Program practices include all processes that may be used 
systematically by an agency or organization to establish and support mentoring relation- 
ships for youth. 

To date, research attempting to delineate beneficial program practices has been concen- 
trated nearly entirely on the issue of effectiveness, as defined above (i.e., impacts on 
mentoring relationship quality and youth outcomes), rather than on other potential indi- 
cators of quality (e.g., safety or efficiency). For this reason, the present article focuses 
on mentoring program practices in relation to issues of effectiveness, while recognizing 
that their implications for program quality conceptualized more broadly is a key concern 
in need of greater investigation. For many reasons that will be explained, the task of 
identifying effective mentoring program practices is a more complex undertaking than 
might initially be expected. There is, however, evidence that effectiveness is far from 
uniform across mentoring programs. For example, in a meta-analysis that my colleagues 
and I conducted of 55 different evaluations of youth mentoring programs (DuBois, Hollo- 
way, Valentine, & Cooper, 2002), about 10 percent showed average effects in a negative 
direction (i.e., youth who received mentoring were worse off), one-third yielded effects 
that were close to zero (i.e., neither positive nor negative), and the remainder exhibited 
positive impacts that ranged in size from small to large. Findings further indicated that 
the typical youth received only modest benefits from participating in a mentoring 
program (average Cohen's d effect size of .18, which would be considered small). 



RESEARCH IN ACTION, ISSUE 2 I 03 



These trends are suggestive of a program effectiveness problem in which there is wide 
variability in the effectiveness of youth mentoring programs and ample room for increases 
in their overall effectiveness as well. 

Fields such as medicine, psychotherapy, and public health have increasingly embraced 
evidence-based practice as an approach for improving the quality of services and pro- 
grams. Applied to youth mentoring, an evidence-based approach would involve using 
research to identify practices that reliably serve to enhance program effectiveness as 
well as any other identified dimensions of program quality. Alternative viewpoints might 
emphasize other strategies for addressing concerns about program effectiveness and 
quality. These include, for example, increased funding for programs to allow for better 
implementation of existing practices, program credentialing so that legitimacy and 
support are offered only to those programs judged to be high in quality, and reliance 
on practitioners or policymakers to identify best practices. Most likely, a combination 
of evidence-based and other approaches will be necessary to improve the level and 
consistency of mentoring program effectiveness. Indeed, it is clearly important that as 
evidence-based practices are delineated: they are supported through adequate funding; 
programs using these practices are reliably identified (as might occur through credential- 
ing); and that the knowledge base continues to grow through testing of practices viewed 
as promising by those in applied roles. 

The following sections of this article provide a brief overview of selected conceptual and 
methodological issues involved with identification of evidence-based mentoring pro- 
gram practices as well as a framework for summarizing and evaluating the evidence that 
pertains to the effectiveness of a given practice. Next, the status and major findings of 
existing research on the efficacy of mentoring program practice are summarized. A more 
in-depth examination of the evidence relevant to a specific practice — pre-match training 
of mentors — is also provided as an illustration of the process that might be involved with 
evaluating the evidence to support the effectiveness of a particular program practice. 

Conceptual and Methodological Issues 

The effects of any practice on mentoring program effectiveness will to some extent be 
dependent on or conditioned by a range of other factors. These include the other prac- 
tices that are being implemented in the same program, the population of youth being 
served, the characteristics and backgrounds of the mentors, the experience levels and 
qualifications of program staff, program infrastructure and financial resources, and the 
broader community context within which the program operates. Illustratively, whereas 
training for mentors could be identified as generally beneficial, this might not be the 
case in a program that lacks adequate follow-up support once mentoring relationships 
are underway, or one in which staff are not well-equipped to offer effective training. 
Closely related to this concern is the possibility that a practice may need to be adapted 
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by host programs in order for its full value to be realized. Continuing with the example 
of training, even a curriculum with strong research support might require modification 
to fit the needs of a particular program based on such considerations as the program's 
goals, expectations and guidelines for mentors, and targeted population of youth. The 
question of how much latitude should be granted for users to make modifications to 
research-supported programs or practices remains controversial in the prevention field 
and is often referred to as the "local adaptation-fidelity debate" (Elliot & Mihalic, 2004). 
Although easy answers will not be at hand, efforts to identify evidence-based mentoring 
program practices always will need to grapple in a thoughtful manner with the question 
of which aspects of practices are essential and which can or should be modified and in 
what ways and under what conditions. 

A final concern is that evidence-based practices once identified are likely to have practical 
utility only to the extent that clear guidelines are developed for assessing their presence 
or absence in a given mentoring program. Ideally, these guidelines would include bench- 
marks that allow for a determination of whether a practice can be considered present 
(cf. Herrera, Sipe, & McClanahan, 2000). Aside from the issue of intentional modifications 
discussed above, this is an important consideration because of the reality that even 
earnest attempts at full implementation of a practice are likely to fall short of ideal levels. 
Consider, for example, with respect to mentor training that staff may fail to cover all 
intended content and that mentor attendance may be less than complete. The depth of 
research required to develop a fully operational set of benchmarks for evidence-based 
practices in youth mentoring is currently lacking, but clearly should be a priority as the 
field moves forward. 

A Framework for Summarizing and Evaluating Research 
on Mentoring Program Practice Effectiveness 

A framework for summarizing and evaluating research on the effectiveness of mentoring 
program practices is shown in Table 1. Hypothetical examples of research studies are 
included as well to illustrate the different types of evidence and findings that together 
comprise the major dimensions of the framework. In actual use, all the research that is 
summarized would pertain to a particular practice of interest; for present purposes, 
examples are provided that pertain to several different program practices. 

A detailed and comprehensive discussion of the proposed framework is outside the 
scope of this article. It will be useful, however, to consider several of its overarching 
features and key assumptions. First, the framework distinguishes three general levels 
of evidence. Level 1 involves study designs that isolate the mentoring program practice 
of interest (e.g., all participants in a program are exposed to the same practices with the 
exception of the practice of interest which only some participants receive). This level 
represents the highest or most conclusive evidence of the impact of a practice on 
program effectiveness. Level 2 involves less precise comparisons in which the practice 
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of interest is not isolated (e.g., the practice is embedded in an overall program which 
is being compared to a no treatment control condition). This level represents a notably 
weaker form of evidence because any differences in outcomes could be attributable in 
whole or in part to other practices with which the practice of interest is bundled. Level 3 
involves qualitative research. These types of studies can be highly informative in many 
ways. Nevertheless, because the design features, quantitative measurements, and sta- 
tistical controls necessary to isolate the effects of a program or practice are typically not 
present (Shadish et al., 2002), qualitative research by itself generally does not constitute 
a reliable indicator of effectiveness in evidence-based practice models. Within each of 
the three general levels of evidence in the proposed framework, two sublevels are fur- 
ther identified (see Table 1). The first of these sublevels in all instances (e.g., la) repre- 
sents the stronger of the two in terms of informing determinations of the effectiveness 
of a practice. 

As described above, the framework focuses on study design features as a primary deter- 
minant of the strength of the evidence made available by any given investigation. It is 
assumed, however, that the full range of other methodological concerns also will be 
taken into account. These include issues relating to sample size and representativeness, 
reliability and validity of measures, and appropriateness of data analytic procedures and 
statistical inferences (DuBois & Silverthorn, 2005). 

Depending on the methodological rigor of a study when considered in this broader 
context, even an investigation with an ideal design could be considered to be seriously 
compromised and its findings thus accorded limited weight or influence. Conversely, a 
study with a relatively weak design (in the context of this framework) would merit greater 
weight if other aspects of its methodology are exemplary. These considerations notwith- 
standing, it is assumed that the degree to which a study's design is well suited to clarify- 
ing the contributions of a practice to program effectiveness is of fundamental importance 
and that deficiencies in this area cannot be fully overcome by other desirable aspects 
of study methodology. So, for example, a Level 2 study typically would not be regarded 
as providing strong evidence of a practice's impact given the inherent imprecision of a 
design that does not isolate the influence of the practice of interest from that of other 
practices. This precedence accorded to study design is consistent with other evidence- 
based frameworks (e.g., What Works Clearinghouse, n.d.). 

A final important feature of the framework is that not only studies in the formal research 
literature, but also internal analyses of information by mentoring agencies and organiza- 
tions (and other stakeholders, such as funders) are assumed to be potentially relevant 
to evaluating the effectiveness of a practice. Examples of the latter type of efforts appear 
in italics within the table. Their inclusion within the framework provides one avenue for 
addressing concerns discussed previously that host programs may have regarding the 
relevance of available evidence to their program and participants and the potential 
need for local adaptations. In instances in which outside research is lacking, agency- or 
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organization-generated research may also be the only source of information available. 

This often may be the case, for example, for novel practices that are not yet widely dis- 
seminated. More broadly, it seems valuable, too, for the field as a whole for mentoring 
programs and organizations to be actively engaged in generating data that can inform 
their program improvement efforts, rather than relying solely on outside sources. 

In a typical application of the framework, the first step would be to identify research that 
is relevant to the mentoring program practice of interest. Next, each study would be cat- 
egorized according to the level of evidence it represents. Its findings then would be sum- 
marized in that row of the framework, taking care to distinguish among positive findings 
(i.e., those that are supportive of the practice enhancing relationship or youth outcomes), 
null findings (i.e., those that fail to reveal any reliable differences in outcomes), and 
negative findings (i.e., those that indicate the practice may detract from effectiveness). 

The final, and arguably most challenging, step would be to synthesize the available 
evidence and make a judgment regarding the practice's effectiveness. This process is 
likely to be complicated by inconsistency in findings both across and within studies. For 
example, some studies may find evidence that matching youth and mentors on the basis 
of race or ethnicity improves mentoring relationship quality, others may fail to find such an 
association, and still others may report mixed findings across different measures of quality 
such as length and emotional supportiveness. In other areas (e.g., public health), expert 
panels of researchers and other stakeholders (e.g., practitioners) have been convened to 
review the available evidence and arrive at joint decisions concerning the evidence base 
to support different practices. Typically, determinations are not "all or nothing" but rather 
make somewhat more fine-grained distinctions. The What Works Clearinghouse (n.d.) of 
the U.S. Department of Education's Institute of Education Science, for example, in as- 
sessing the strength of evidence for an intervention, provides a rating of positive effects, 
potentially positive effects, mixed effects, no discernible effects, or potentially negative 
effects. These ratings may be provided separately for different types of outcomes (e.g., 
academic and behavioral). Also included are an Improvement Index that gives a sense 
of the size of the intervention's effect and an Extent of Evidence rating that indicates how 
much evidence was used to determine the intervention rating. 

Research on Program Practices 

Descriptive Studies 

Before considering research that addresses the impact of different mentoring practices, 
it is worth noting the value of investigations with more descriptive aims. These studies 
may highlight important trends in program practices that merit further investigation. A 
national survey of 1,762 mentoring programs conducted by MENTOR/National Mentoring 
Partnership (Manza, 2003), for example, revealed that site-based programs had nearly 
eclipsed more traditional community-based programs in popularity, thus underscoring 
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a need for research on site-based approaches. Descriptive studies also can provide infor- 
mation concerning the extent to which practices with evidence-based support are being 
adopted and in what ways they are being adapted by programs in the process. In the 
same survey referred to above, for example, nearly a quarter of the programs surveyed 
(24 percent) did not report providing pre-match orientation and training to mentors. 
Another survey of more than 700 mentoring programs (Sipe & Roder, 1 999) found that 
among those programs that did require mentors to attend an orientation and complete 
some type of pre-match training, the total amount of time spent in these activities varied 
greatly (Sipe & Roder, 1999). In a field that is evolving and expanding rapidly, such as 
youth mentoring, there will always be a need for high-quality descriptive studies of 
current program practices. 

Status and Major Findings of Research on the Effectiveness of Program Practices 

Numerous mentoring programs have had their effectiveness for promoting youth outcomes 
examined in rigorous evaluation research. These studies are of relatively limited value, 
however, for informing understanding of the importance of particular program practices. 
This is because, as noted previously, the typical program evaluation is designed to estimate 
the cumulative effects that are produced by the multiple practices that comprise a given 
program rather than the unique contribution of any one practice (i.e., Level 2a and 2b 
in the evidence framework described above; see Table 1). There still may be substantial 
unplanned variation in exposure to a particular practice across participants in the evalua- 
tion of a mentoring program (DuBois et al., 2006). This type of variation may occur for a 
variety of reasons, including differences in implementation across both program sites (in 
the case of multi-site studies) and staff within sites. Analyses of the association between 
such differences and outcomes may be informative with regard to the practice's contri- 
butions to program quality (i.e., Level 1b evidence). Unfortunately, opportunities of this 
nature have not been exploited consistently in evaluations of mentoring programs. 

The most extensive empirical examination of program practice effectiveness to date was 
carried out as a part of the meta-analysis referred to previously that my colleagues and 
I conducted (DuBois et al., 2002). In analyses that controlled for the methodological 
characteristics of studies, we found that seven different mentoring program practices 
were predictive of stronger positive effects on youth outcomes: procedures for system- 
atic monitoring of program implementation, use of community settings (as contrasted 
with only the school setting) for mentoring, utilization of mentors with backgrounds in 
helping roles or professions, clearly established expectations for frequency of mentor- 
youth contact, ongoing (post-match) training for mentors, structured activities for men- 
tors and youth, and support for parent involvement. The magnitude of estimated effects 
increased systematically as programs utilized a greater number of the practices, thus sug- 
gesting that they made independent contributions to youth outcomes. These findings 
offer a noteworthy level of evidence-based support for the practices involved (i.e., Level 
1 b in the evidence framework presented in this article). The program practices them- 
selves, however, were only able to be coarsely defined in this research. It is therefore not 
clear what types of delivery formats, content, or levels of intensity may be required for 
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any of the practices involved to contribute positively to youth outcomes. The less than 
optimal level of specificity in how program practices were able to be assessed also under- 
scores that it would be inappropriate to conclude that other practices not predicting dif- 
ferences in effectiveness in this research are necessarily unimportant (DuBois et al., 2002). 



Pre-Match Training 

In the meta-analysis, programs that provided pre-match training to mentors did not have 
significantly greater estimated effects on youth outcomes (DuBois et al., 2002). Several 
studies have also examined associations between naturally occurring variations in expo- 
sure to initial training within programs and indices of relationship quality (Furano, Roaf, 
Styles, & Branch, 1993; Herrera et al., 2000; Herrera, Grossman, Kauh, Feldman, & Mc- 
Maken, 2007; as with the meta-analysis, the findings of these studies provide evidence at 
the 1 b level in the proposed evidence framework). In a study of 669 mentors who were 
involved either in one-on-one community-based or school-based programs (Herrera et 
al., 2000), those mentors who indicated they had attended fewer than two hours of pre- 
match orientation or training reported the lowest levels of closeness and supportiveness 
in their relationships with their mentees, whereas those mentors reporting the strongest 
relationships had attended six or more hours of orientation or training prior to the match. 
These associations were evident for mentors in both types of programs. In the recent na- 
tional impact study of Big Brothers Big Sisters school-based mentoring programs (Herre- 
ra et al., 2007), mentors who reported receiving more pre-match training reported higher 
levels of efficacy, or confidence, before being matched and were more likely to extend 
their relationship into a second school year. Those mentors who reported receiving 
more individual training pre-match also reported having closer relationships with their 
mentees. Not all findings in this study, however, supported the importance of pre-match 
training. Whether the mentor participated in training and the amount of group training 
received, for example, were not significantly associated with either the length or men- 
tor-reported closeness of relationships that developed during the first year of program 
involvement. Youth reports of relationship closeness (as opposed to mentor reports), 
furthermore, were unrelated to any of the pre-match training measures. In another study 
examining training across eight agencies, Furano et al. (1993) found that indicators of 
relationship quality (average number of meetings, percentage of matches not meeting, 
matches not meeting because of loss of interest) did not differ significantly between four 
agencies that provided pre-match training (three hours, on average) and the remainder 
that did not. 

In a further noteworthy study (Cavell & Hughes, 2000), 62 teacher-identified aggressive 
second- and third-grade children were assigned randomly to receive either "therapeu- 
tic" or "standard" mentoring. Mentors in the therapeutic condition received 18 hours of 
training to enhance their understanding of childhood aggression and their 
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capacity to develop and maintain an emotionally supportive relationship with an aggressive 
child; those in the standard condition received only a one-hour orientation, focusing on 
safe and appropriate mentoring activities. The two conditions were further distinguished 
by the mentors of children in the therapeutic condition receiving regular supervision and 
having a longer period of time to establish relationships with children (children in the 
standard condition were assigned new mentors each semester) and by children in this 
condition receiving problem-solving skills training and parent and teacher consultation. 
The study design thus did not isolate the practice of interest (pre-match training). It did, 
however, feature an experimental contrast involving this practice and a well-defined set of 
additional program elements (i.e., Level 2a evidence; see Table 1). Overall, findings failed 
to reveal any differential effects of the therapeutic mentoring condition on youth outcome 
measures (e.g., aggressive behavior). Children in the therapeutic mentoring condition did 
report viewing their mentors more positively than those receiving standard mentoring. 
There also was some evidence that the therapeutic mentoring condition was a better fit 
than standard mentoring for those children who had experienced parental rejection or 
who were viewed negatively by their peers (Cavell & Hughes, 2000). 

Qualitative investigations also have reported findings that address the potential role of 
training. In one notable investigation (Spencer, 2006), in-depth semi-structured interviews 
were conducted with 24 adolescent and adult pairs who had been in a continuous men- 
toring relationship for a minimum of one year and then were systematically analyzed (i.e., 
Level 3a evidence; see Table 1). Findings suggested that it could be beneficial to provide 
training for mentors that addresses cultural issues pertaining to class and race differences 
in a more intensive manner than has been the case typically in programs (Spencer, 2006). 
A review of primarily qualitative research on mentoring in after-school programs (Hirsch & 
Wong, 2005) similarly concluded that enhanced training for staff could help them to more 
fully realize their potential to provide effective mentoring to youth. 

Overall, the available evidence regarding the effectiveness of pre-match training for 
mentors is mixed. In an actual application of the proposed framework, a panel of re- 
searchers and practitioners ideally would convene to consider this evidence and poten- 
tial explanations for the variability in results. Along with the methodological quality of 
studies, it would be important to consider the quantity and delivery format of the train- 
ing provided in each investigation as well as the content of training materials and their 
likely value in helping mentors build beneficial and enduring relationships with the youth 
being served by the program(s) involved. This process could yield collective informed 
judgments regarding the likely effectiveness of pre-match training, associated recom- 
mended benchmarks, and the strength of the available evidence. Decisions regarding 
the use or endorsement of this practice by a mentoring agency, organization, or other 
stakeholder (e.g., funder), in turn, could then be made with appropriate consideration 
given to other pertinent factors. These include the relative strength of evidence-based 
support for other program practices (e.g., post-match training) that might serve similar 
needs as well as the goals and resources of the mentoring programs involved and the 
practices they already have in place. 
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Conclusions 



The framework presented in this article represents a first step toward developing a sys- 
tematic approach for integrating and evaluating evidence regarding the effectiveness of 
different mentoring program practices. To date, the foundation of research that is re- 
quired to make informed judgments using this framework is in an early stage of develop- 
ment. For purposes of illustration, this article reviewed the evidence base for pre-match 
training. Other practices that are widely regarded as core components of mentoring 
programs, such as volunteer screening, matching, and supervision, have received a 
similar degree of preliminary study and thus are also in need of more extensive investiga- 
tion. Still other basic program functions, such as recruitment, have been largely ignored 
by researchers and will require even more dedicated investments. Newer modalities and 
structures for mentoring (e.g., group, site-based, peer, on-line), most of which have only 
just begun to be studied in earnest, clearly should receive more in-depth examination 
as well (Rhodes & DuBois, 2006). These considerations underscore the importance of 
coupling application and ongoing refinement of the proposed framework with intensi- 
fied efforts to strengthen the available evidence base. These complementary directions, 
if embraced by the field, hold significant promise for yielding insights and recommenda- 
tions that substantially improve mentoring program effectiveness. 



RESEARCH IN ACTION, ISSUE 2 I 11 



Table la: Framework for Characterizing the Evidence for Mentoring Program Practices 



ir. i 1 


Examples 01 


LviutJiiLt? i_evei 


L^eULI ipilUl 1 


Positive Finding 


Null Finding 


Negative Finding 


la: Experimental 
Test of Practice of 
Interest (POI) 


Research com- 
paring outcomes 8 
with and without 
the practice of 
interest (POI) 
using an experi- 
mental design bc . 


A study randomly 
assigns mentors in 
a program to re- 
ceive or not receive 
pre-match training 
and finds favorable 
effects of training 
on relationship 
quality and youth 
outcomes. 


An agency 
randomly assigns 
mentors to receive 
or not receive post- 
match training and 
finds no difference 
between groups 
in mentoring 
relationship quality 
or youth outcomes. 


A study randomly 
assigns programs 
to use or not use 
a new training 
protocol for 
mentors and finds 
a negative effect 
of the new training 
on mentoring 
relationship quality 
as rated by youth. 


1 b: Quasi- 
experimental 
Test of POI 


Research com- 
paring outcomes 3 
with and without 
the POI using a 
quasi-experimental 
design (includes un- 
planned variations 
in exposure to POI 
within studies as 
well as comparisons 
across studies using 
meta-analysis) b . 


An agency 
compares youth 
outcomes prior 
to and after 
instituting group 
" get acquainted" 
activities to 
facilitate volunteer- 
youth matching 
and finds improve- 
ments in youth 
outcomes. 


A study comparing 
mentors who 
did and did not 
receive the required 
number of support 
contacts from 
agency staff finds 
no difference 
in relationship 
closeness or 
duration between 
the two groups. 


A meta-analysis 
finds that programs 
offering mutual 
support groups 
for mentors have 
higher percentages 
of relationships 
that terminate 
prematurely. 


2a: Experimental 
Test of Program/ 
Set of Practices 
that Includes POI 


Experimental 
research compar- 
ing outcomes 3 
for a program 
(or collection of 
practices) that 
incorporates the 
POI and a no- 
treatment control 
condition or other 
condition that does 
not include POI c . 


A study randomly 
assigns youth to 
either a mentoring 
program that 
includes a 
curriculum of 
group activities 
to promote 
relationship 
development or 
to a no-treatment 
control group 
and finds positive 
effects of the 
program on 
both relationship 
quality and youth 
outcomes. 


A foundation 
randomly assigns 
programs to 
receive or not 
receive funding to 
support program 
enhancements and 
finds no difference 
between the two 
groups of programs 
in relationship 
quality or youth 
outcomes. 


A study assigns 
programs to 
use a collection 
of promising 
"enhanced" 
mentoring practices 
or to continue using 
standard practices 
and finds poorer 
relationship quality 
in programs using 
the newer practices. 
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Table 1b: Framework for Characterizing the Evidence for Mentoring Program Practices 









Examples d 








Positive Finding 


Null Finding 


Negative Finding 


2b: Quasi- 
experimental Test 
of Program/Set 
of Practices that 
Includes POI 


Quasi-experimental 
research comparing 
outcomes for 
programs (or collec- 
tions of practices) 
that incorporate 
the POI (includes 
unplanned variations 
in exposure to the 
POI within studies 
as well as relevant 
comparisons across 
studies using meta- 
analysis). 


A study compares 
youth who either 
did or did not 
participate in a 
school-based 
mentoring program 
that includes 
provisions to 
facilitate summer 
contact between 
mentors and 
youth and finds 
more favorable 
outcomes for youth 
participating in 
the program. 


A study compares 
programs that 
do or do not 
conduct home 
visits as part of the 
screening process 
for prospective 
mentors and finds 
no difference 
between the two 
types of programs in 
relationship quality. 


An agency 
compares outcomes 
before and after 
adopting a new set 
of practices resulting 
from a strategic 
planning process 
and finds a decrease 
in relationship 
quality or youth 
outcomes. 


3a: Qualitative 
Investigation 


Qualitative 
research of 
multiple programs 
or program 
participants in which 
data relevant to the 
POI are collected 
and analyzed 
(includes narrative 
literature reviews). 


A national 
mentoring 
organization 
conducts focus 
groups with 
mentoring program 
administrators and 
findings suggest 
that adoption of 
the organization's 
practice guidelines 
improves mentoring 
relationship quality. 


A literature review 
concludes that 
matching youth and 
mentors on race/ 
ethnicity does not 
make a difference in 
relationship quality 
or youth outcomes. 


Mentoring 
relationships that 
have terminated 
prematurely are 
investigated 
through interviews 
with youth and 
mentors and 
findings suggest 
that a lack of 
regular supervision 
contributes to 
relationships ending 
prematurely. 


3b: Case Study 


Case studies of 
individual programs 
(or individual 
participants 
or mentoring 
relationships in 
programs) in which 
data relevant to the 
POI are collected 
and analyzed. 


A case study of a 
mentoring program 
suggests that 
matching youth and 
mentors on ethnic 
background may 
promote better 
youth outcomes. 


A case study of 
an after-school 
program suggests 
that the practice 
of incorporating 
mentoring into 
the program's 
mission statement 
does not by itself 
contribute to the 
development of 
beneficial mentoring 
relationships 
between youth 
and staff. 


A staff case 
presentation of a 
youth participating 
in a mentoring 
program suggests 
that the practice 
of adopting goals 
focused exclusively 
on increasing the 
number of youth 
served may detract 
from the quality of 
the relationships 
being established in 
a program. 



a Outcomes in the context of this framework include indicators of both mentoring relationship quality and 
youth adjustment. b The design must be one in which the only practice that differs between conditions or 
groups is the POI. c Experimental designs involve random assignment to different conditions; alternative 
designs that provide a similarly strong basis for making determinations of causal influence ( e.g ., regression 
discontinuity) also are included in this level of evidence (see Shadish, Cook, & Campbell, 2002, for a 
discussion of these types of designs). d AII examples are hypothetical and thus do not reflect actual studies 
that have been conducted. 
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ACTION 



Effectiveness of Mentoring Program Practices 



Like many young disciplines, innovation in the practice of mentoring far outpaces available 
research on the effectiveness of mentoring. Mentoring programs are often forced to 
adapt to changing communities, youth populations, and volunteer motivations without 
the benefit of formal research. This often results in confusion about what works and why, 
and may lead to the perpetuation of ineffective practices. In his article, Dr. DuBois out- 
lines a framework for evaluating mentoring program practices by delineating the various 
evidence levels required. Importantly, the framework not only includes formal research, 
but also incorporates internal analyses of information by mentoring programs and organi- 
zations. Most mentoring programs lack the resources to conduct formal research and 
find it difficult to decipher broader research findings into practical implications for their 
programs. This action section outlines a step-by-step process that agencies and organiza- 
tions can use to design and conduct evaluations of specific practices within their programs. 



How to Evaluate Program Practices 

There are many types of program evaluation assessing everything from program process- 
es to youth outcomes. The steps below describe how mentoring programs can evaluate 
the effectiveness of a specific practice. 

Step 1: Identify a problem, challenge or opportunity your program is facing. 

Example: A number of mentors have expressed concerns about their mentees' 
lack of enthusiasm for the program and unrealistic expectations for their 

relationships. 

Step 2: Develop a hypothesis or prediction regarding a practice that addresses 
the problem. 

Example: Mentees who receive training prior to being matched will be better 
prepared for mentoring, resulting in longer-lasting relationships. 

Step 3: Determine what practice you need to asses to test your hypothesis. 

Example: Mentee training 
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Step 4: Formulate the research question you want to answer. 

Example: Will providing initial training for mentees increase match duration? 

(if no mentee training is currently provided) Or, does providing ongoing training 
for mentees increase match duration (if only initial mentee training is provided)? 

Step 5: Design the evaluation. 

a) Decide what your comparison group will be and keep in mind the varying 
levels of Dr. DuBois's framework. 

Example 1 (Evidence Level la): The program randomly assigns mentees in the 
program to receive or not receive training and compares the average match 
duration of the two groups. 

Example 2 (Evidence Level 1b): The program initiates training for all new 
mentees and compares average match duration one year after the trainings 
began to average match duration before the trainings began. 

b) Decide how you will measure the outcome(s). Where will the information come 
from? How will it be collected? 

Example: Use case management data to determine the duration of each 
match in both the control and training groups. 

Step 6: Collect and analyze the data. 

Example: Determine the average relationship duration for the control and 
training groups and compare. 

Step 7: Use the results. 

Example 1: The analysis shows that the relationships of mentees who received 
training lasted three months longer, on average, than those who did not. Make 
mentee training a standard part of the program. 

Example 2: The analysis shows no difference in relationship duration between 
the group of mentees who received training and those who did not. Assuming 
the training was implemented as intended, go back to step 2 and develop 
another hypothesis. 
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RESOURCES 



Effectiveness of Mentoring Program Practices 



Information Technology International (ITI). ITI provides professional, programmatic, 
and technical support to government and industry, www.itiincorporated.com 

• Evaluating your Program: A Beginner's Self-Evaluation Workbook for Mentoring 
Programs. 

www.itiincorporated.com/_includes/pdf/SEW-Full.pdf 

MENTOR/National Mentoring Partnership. The leader in expanding the power 
of mentoring to millions of young Americans who want and need adult mentors. 
www.mentoring.org 

• The Elements of Effective Practice™ 

www.mentoring.org/program_staff/design/elements_of_effective_practice.php 

• How to Build a Successful Mentoring Program Using the Elements of Effective 
Practice™ 

www.mentoring.org/eeptoolkit 

• How to Select a Survey to Assess Your Adult- Youth Mentoring Program 
www.mentoring.org/program_staff/eeptoolkit/evaluation/howtoselectasurvey.doc 

Northwest Regional Educational Laboratory - National Mentoring Center. National 
organization that provides training and technical assistance to youth mentoring 
programs, www.nwrel.org/mentoring 

• Frequently Asked Questions About Research and Evaluation 
www.edmentoring.org/pubs/ws2_supplement2.pdf 

• Measuring the Quality of Mentor Youth Relationships: A Tool for Mentoring Programs, 
Technical assistance packet. 
www.nwrel.org/mentoring/pdf/packeight.pdf 
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